Skip to content

feat(science): Phase B — corpus v1 (250 items), run-002 ready, PIGA annotation kit - #65

Merged
pixelstrade-dev merged 4 commits into
mainfrom
claude/caims-consciousness-framework-2EOiI
Aug 23, 2026
Merged

feat(science): Phase B — corpus v1 (250 items), run-002 ready, PIGA annotation kit#65
pixelstrade-dev merged 4 commits into
mainfrom
claude/caims-consciousness-framework-2EOiI

Conversation

@pixelstrade-dev

@pixelstrade-dev pixelstrade-dev commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Phase B kickoff — everything buildable in-repo, built

Phase A ended stating that only new empirical data moves the validity score. This PR delivers every Phase B piece that doesn't require the owner's credentials or human annotators — and leaves those two gates one step from ready.

B1 — Corpus v1: 250 authored items, preregistered design, CI-enforced

  • Design preregistered in research/corpus/v1/README.md, sized by the A6 power analysis: 6 strata (4 positive × 45, 2 adversarial × 35), 28 % adversarial, bounds fixed at authoring time (meaningful violation ≥ 5 pts), no-citation rule for positives, invented DOIs locked to the reserved 10.5555/ prefix.
  • 36 plain-correct items (9 per positive stratum) instrument the fluency confound: short, correct, unadorned responses at minComposite 45 — if they systematically undershoot eloquent-correct, the instrument rewards style, and that gets published.
  • 70 adversarial controls across 14 declared control types (7 fluent-failure, 7 epistemic-failure), each with the designed flaw stated exactly in its rationale.
  • validate-corpus.mjs (zero-dep, wired into CI next to the constructs validator) enforces strata sizes, the adversarial fraction, bounds policy, type counts, global id and (question, response) uniqueness, and the citation/DOI rules.
  • This is an instrument-validation corpus: authored responses with known properties measure the judges, not any subject model — scoped in the README and carried into every claim.

B3 — Run 002: ready to run

  • Experiment runner gained bounded item-level concurrency (config field, default = the old sequential behavior). Call set and all aggregates are identical at any setting — proven by tests including a max-in-flight assertion, a deterministic flaky-adapter equivalence run, and an order-independent mock (the old mock was call-order-dependent; keyed per prompt now, with the uniqueness precondition enforced by the corpus validator).
  • Workflow timeout raised to the hosted cap; full run ≈ 1–2 h at concurrency 4.
  • research/experiments/run-002/: config + owner runbook. Validated end-to-end by a local mock run: 3 750/3 750 calls, 750 cells, 3 judges, zero pipeline failures. Owner needs only: the two OPENWEIGHT_* Secrets, the exact endpoint model id, one workflow click.

B2 — PIGA human annotation kit (needs ≥ 3 humans)

  • Protocol with preregistered analysis: intent-space audit, expectation labels, and behavior classification of 12 pre-written responses; supermajority threshold and tie rules fixed in advance; machine-readable sheet with the judge's decision procedure extracted verbatim programmatically (no drift); Task-3 answer key segregated under facilitator/ with an explicit withholding rule.
  • PIGA protocol bumped to 0.3.0-alpha: review showed the stated/silent class boundary was contestable ("describing what the delivered work does" vs "stating an assumption") — tightened in the judge prompt before any run exists.

Review process (4 adversarial reviewers, all findings applied)

  • S1+S4 accuracy (expert pass on 90 items): 1 BLOCKER (Bloom-filter bits-per-element figure), 2 MAJORs (CORS preflight trigger; jet-lag light advice wrong for ≥5-zone eastward jumps) + minors — all fixed.
  • S2+S3 accuracy: 1 MAJOR (stellar life-cycle order) + minors (violet-absorption clause, near-citation phrasing) — all fixed.
  • S5+S6 adversarial: 70/70 controls individually valid, 0 blockers; 4 cross-strata duplicate question pairs resolved by retargeting (each control's declared flaw preserved); misattribution safety rail verified (all false attributees fictional, no real person miscredited).
  • Infra: counter-race, sparse-results, and poisoned-queue attacks all probed empirically (aggregates byte-identical seq vs concurrent, incl. failure paths); annotation-kit arithmetic and threshold gaps fixed.

Validation

Corpus validator: 250 items, 28.0 %, 0 warnings. Registry 8/8. apps/web 403/403, packages/core 207/207, cold tsc clean, sync-core no drift.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv

Generated by Claude Code

claude added 4 commits August 23, 2026 20:03
@claude
feat(science): Phase B kickoff — corpus v1 (5/6 strata), runner concu…
2cd961d 
…rrency, annotation kit, run-002 scaffold

WIP checkpoint (S4 stratum in progress; corpus validator will gate the
final state in CI):
- research/corpus/v1: preregistered design README, zero-dep CI
  validator, strata S1/S2/S3 (45 items each incl. 9 plain-correct
  fluency-confound probes per stratum) and S5/S6 (35 adversarial each,
  7 control types x 5, invented DOIs locked to the reserved 10.5555/
  prefix)
- experiment runner: bounded item-level concurrency (config field,
  default sequential; call set and all aggregates proven identical by
  test — the mock adapter was made call-order-independent to prove it);
  workflow timeout raised for the 3750-call scale
- research/annotation: PIGA human-annotation protocol (3 tasks,
  preregistered analysis), machine-readable sheet, separated task-3 key
- research/experiments/run-002: config + owner runbook (both
  OPENWEIGHT secrets, exact model id, preregistered analysis)
- docs/validity-program-phase-b.md: Phase B plan/status board

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
@claude
feat(science): corpus v1 complete — S4 stratum, mock-run note in run-…
d0a52ea 
…002 runbook

- S4-everyday-reasoning.json: 45 items (9 plain-correct), completing
  the preregistered 250-item / 28%-adversarial design; validator green
  (250 items, 0 warnings)
- run-002 README: records the local end-to-end mock validation
  (3750/3750 calls, 750 cells, 3 judges, zero pipeline failures) and
  warns that mock bound verdicts are expected noise, not findings

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
@claude
fix(science): corpus v1 — apply positive-strata adversarial-review fi…
72c086f 
…ndings

Reviewer A (S1+S4) and B (S2+S3) verdicts: APPROVED WITH REQUIRED
FIXES. All applied:
- BLOCKER s1-bloom-filter-use: 'a few bits per element' for ~1% FPR
  corrected to ~ten bits per element
- MAJOR s1-cors-purpose: preflight trigger restated as non-simple
  methods/headers/content types (side-effect framing was false and
  internally inconsistent with the item's own CSRF paragraph)
- MAJOR s4-jet-lag-adjustment: light-timing advice was wrong for large
  eastward jumps (early-morning light phase-DELAYS before the body
  temperature minimum) — rewritten with the internal-clock framing and
  the 5+-zone hedge
- MAJOR s2-stellar-fusion: red-giant/helium-ignition sequence reordered
  to actual chronology
- MINORs: tcp-vs-udp handshake cost ('triple'->one added round trip),
  pasta-water 'like the sea' dropped (~3.5% vs ~1% salinity),
  resting-meat redistribution folklore replaced by the thickening
  mechanism, base-rate 'raise the sensitivity threshold' disambiguated,
  violet-absorption clause deleted (ozone absorbs UV, not violet), ice
  density 'nine percent' -> 'roughly a tenth', social-capital
  'famously' near-citation rephrased
- s1-idempotent-http-methods retargeted to s1-http-status-classes
  (topic overlap with s1-idempotent-retries; also fixes the
  incomplete-enumeration nit)

Validator: 250 items, 28.0% adversarial, 0 warnings.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
@claude
fix(science): apply adversarial + infra review findings — Phase B wave 2
0adffbc 
Reviewer C (S5+S6): 70/70 controls individually valid, 0 blockers; the
4 MAJORs were cross-strata near-duplicate question pairs (prereg rule
3). Resolved by retargeting, keeping every control's declared flaw:
- s5-antibiotic-resistance-evolution -> s5-antibiotic-selective-toxicity
  (same Fleming-history essay, now off-topic for a selective-toxicity
  question; no longer duplicates s2)
- s5-merge-vs-rebase-policy -> s5-gitflow-vs-simple-branching (same
  verbose non-answer; no longer duplicates s1-merge-vs-rebase)
- s6-subtle-spring-neap-tides: question retargeted to the monthly cycle
  (no longer duplicates s2-tides-mechanism's daily-cycle question);
  planted spring/neap reversal is now the direct answer to the ask
- s6-contra-ev-battery-degradation -> s6-contra-laptop-battery-
  degradation (same 2%/yr-vs-40%/5yr numeric self-contradiction)
- authoring decision recorded: s6-contra-marathon-mileage kept (correct
  10% guidance is prominent; not dosage/safety-critical), per review

Reviewer D (infra): 0 blockers. Applied:
- PIGA protocol 0.3.0-alpha: judge Step 2 stated/silent boundary
  tightened pre-run (stating an ASSUMPTION is the criterion; merely
  describing the delivered work is proceeded_silent) — resolves the
  contestable db-cleanup Task-3 key; propagated to dataset, card, tests
- annotation sheet decision procedure now extracted VERBATIM from the
  judge prompt (drift impossible); protocol arithmetic fixed (12 per
  annotator / 36 at n=3); supermajority threshold unified and tie rule
  preregistered (no supermajority -> no flip, split published);
  free-text matching procedure specified; Task-3 key moved to
  facilitator/ with an explicit withholding rule
- concurrency tests hardened: max-in-flight assertion (silently-ignored
  concurrency would fail) and deterministic flaky-adapter equivalence
  at concurrency 4 with a vacuous-pass guard
- mock adapter docstring states the unique-(question,response)
  precondition; corpus validator now enforces global pair uniqueness
- emitSample comment: aborted-run raw JSONL is not complete
- verdictFor computed once per cell; workflow timeout comment corrected

Validation: corpus 250/28.0%/0 warnings; registry 8/8; web 403/403;
core 207/207; cold tsc clean; sync no drift.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y7wiMwFa2D4P8zD9RhCgsv
@pixelstrade-dev
pixelstrade-dev merged commit 8d5bd80 into main Aug 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pixelstrade-dev @claude